x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector#49614
x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector#49614efd6 merged 4 commits intoelastic:mainfrom
Conversation
🤖 GitHub commentsJust comment with:
|
Vale Linting ResultsSummary: 2 suggestions found 💡 Suggestions (2)
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🔍 Preview links for changed docs |
4cbe48d to
78c339c
Compare
|
re: vale linter
|
f7641f4 to
d2a44e8
Compare
Implement the Redirector interface on httpjson's InputManager so that when run_as_cel is true and cel.program is present, the Loader redirects to the cel input with a translated config. convertHttpjsonToCel maps the full set of shared fields between the two inputs: interval, auth, request.url to resource.url, retry, redirect, keep_alive, tracer, and inline transport settings (ssl, timeout, proxy_url, proxy_headers, proxy_disable, idle_connection_timeout). cel-namespaced fields (program, state, regexp, xsd, max_executions, redact) are extracted to their top-level equivalents. Fields specific to httpjson's declarative config (method, body, transforms, response, cursor, chain) are excluded since the CEL program handles them directly. max_executions uses Int/SetInt to match its *int type in the cel config rather than round-tripping through strings.
Read the httpjson cursor from the persistent store during Redirect and inject it into the translated cel config's state.cursor field. The cel input uses config.State["cursor"] as the initial value when no stored cursor exists, so the first run after redirect continues from where httpjson left off. Subsequent runs use the cel-stored cursor and ignore the injected value. Direct store writes are not viable because the Loader calls Init on all plugin managers (loading the cel cursor manager's in-memory table) before any Configure call where Redirect runs. A cel:: key written to the persistent store during Configure would not appear in the in-memory table, and findOrCreate would overwrite it with an empty resource on first flush. The httpjson store entry is preserved for rollback safety.
Add reference documentation for the run_as_cel option and the cel.program, cel.state, cel.max_executions, cel.regexp, cel.xsd, and cel.redact configuration fields. Includes a field mapping table, a note on cursor state migration, and a worked example.
d2a44e8 to
f5b8809
Compare
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a redirect mechanism that converts httpjson inputs into cel inputs when ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Parse the URL through url.Parse before constructing the cursor store key so that the key matches source.Name() normalization. Without this, URL round-tripping through net/url could cause a mismatch between the key we look up and the key the input wrote. Add input.Name() assertion to TestRedirect_EndToEnd to verify the Loader actually redirected to the cel plugin.
Proposed commit message
Note
Best reviewed commitwise.
Checklist
stresstest.shscript to run them under stress conditions and race detector to verify their stability../changelog/fragmentsusing the changelog tool.Disruptive User Impact
Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs